optic <A*_ ronw no. io 
M A.Y IS'T FuntGN 

csa PrTft (u cm) 101*1 1. a 


UNITED STA4fRB r ©(9tfjfi9K 2003/04/29 : CIA-RDP83-00058R0001 001 20007-6 


Memorandum 


OISt8043 


to : Director, OTR 

STAT 

FROM : 


date: 20 July 1973 



subject. Program Performance Measurement. 

The Emerging Role of OTR in establishing 
Requirements for Training. 


I am submitting this memorandum intended as a discussion of 

(1) how to provide the data base necessary to tell OTR how well 
tht, programs are meeting their objectives on a current basis 

(2) an approach to determine to what extent the knowledge and/or 
skills imparted are applied on the job and 

I how OTR can establish requirements as contrasted with 

I historically reacting to requirements. 

A concept of a data base is presented which is intended to aggregate 
all of this and also attached is a basic form that can be used to 
respond to all of these goals. 

In order to institute the best type of (program) performance 
measurement a number of aspects must be considered which not only influence 
the behavior and morale of those being measured but also affect the ability 
to satisfy the student — our custoraer. 


t 



I* 

i:\ 




This paper is divided into four sections 

Section 1 - A Discussion of Program Performance Measurement 
Section 2 - 0 TR t s Role in Establishing Requirements 
Section 3 - Some Observations 
Section 4 - Appendix A ~ C 

STAT 

unairman, Functions Course 
ilry US. Savings Boa :1s Regal arty on the Payroll Savings Plan 
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A discussion of Program Performance Measurement 

There are four steps involved in Program Performance Measurement 
that must be followed in a continually repetitive sequence to achieve 
the full benefits of such efforts. 

The first step requires the careful specification and analysis of 
basic program objectives in each major area of activity. To accomplish 
this one must back away from the particular program being carried on, 
look at their objectives, and ask what are we really trying to 
accomplish. This definition of goals to be accomplished should start 
ideally at the top of the "organization" so that each level can be 
certain that their definition of goals falls within the scope of what 
has been defined by the next higher level. OTR certainly must 
provide the training necessary for agency personnel to accomplish or 
improve the accomplishment of the various agency missions but such 
definition is too broad and must be narrowed. At the other extreme 
for OTR to say we shall train "x" number of people or increase our 
student through-put by some quantity is too narrow and must be 
broadened. Specification of OTR objectives must fall between these 
two extremes. Thu more we learn about how to reach an objective, 
the more clearly we understand the objective resulting in a constant 
interaction between the decision process and our knowledge of our 
true objectives. 

The second step iiwolves the analysis of the output of a given 

program in terms of the objectives initially specified in the first 

step. For the OTR programs, our output addresses not how many or 

how many more students did we turn out but rather what improvements, 

1 
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knowledge and/or new skills have we provided to the student. Will 
our educational efforts influence his behavior on the job and improve 
his job performance. This is often referred to as feedback a term 
taken from engineering servomechanism theory. 

The third step calls for measurement of the total costs of the 
program — not just for one year but over at least several years 
ahead. This would require OTR to identify the resources expended 
in each program, to accumulate resource costs by program and to 
extrapolate into the future periods from a historical cost data 
base. 

The fourth step involves the analysis of alternatives if and 
only if programs are competing for limited resources. As an 
expository application let us examine how these, steps or this cycle 
of events applies to the Information Science Program and the 
Information Science for Intelligence Functions Course within that 
program. 

Course objectives serve two purposes: 

1. They express the desired results of our customers in 
terms of accomplishments or goals to be achieved by 
the course. 

2. They provide the basis or elements fundamental to course 
evaluation. 

The most difficult aspect of any professional performance 
measurement scheme is not how to measure but what should be measured. 
What should be measured is dependent upon how one is organized and 
whether accountability and responsibility for activities which 
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satisfy customers needs are clearly defined and understood by those 
charged with such responsibility and accountability. Clearly 
organizational analysis is beyond the scope of this paper nevertheless 
the interrelationship between the design of an organization and an 
evaluation of an organization's performance must be clearly established. 

These ideas have been used in our Information Science Program. 

For example I have assumed from our previous planning that the single 
overall objective required by step one of our Information Science 
Training Program at the Chief /Information Science Training Staff level 
is to educate Intelligence Professionals in the Information Science 
Disciplines. For the Current FY 73 Community Program segment, the 
numbers of students planned were as follows: 


2 offerings 

— 

Survey — 

3 weeks — 

30‘ 

stdts = 60 

2 offerings 

— 

Functions — 

4 weeks — 

25 

stdts = 50 

2 offerings 

— 

Inf. Sci. — 

1 week — 

25 

stdts = 50 




Total Students 


160 


Total Course Weeks 16 

Assuming for step two that the proper output measure of the 
Information Science Program objective is to quote numbers of students 
exposed to Information Science then I could certify for the Functions 
Course that 27 + 30 or 57 students completed the Functions Course 
during FY 73. One can further document the specific inputs (and 
their costs required in step three) required to generate this output 
of 57 students i.e., lecture time of resident and guest faculty, 
course preparation time, computer time and charges (by problem and 
lecture if necessary) etc. From a Systems Analysis point of view, 

- 4 - 
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the output (57 students) is tied to the related and required inputs 
(the resources and their costs) but is this a sufficient and proper 
measure? This measure of output as a single measure reflects the 
number of students put through the course but does not reflect the 
filVL range of vaLu.es or l>enefvts provided to our students and their 
respective organizations. In order to measure the value provided 
by the Functions Course and its contribution to the Information 
Science Program overall, it is necessary to go beyond the single 
measure of student output and resource input to determine whether 
we accomplished the course objectives and accordingly satisfied 
the students our customers. What is missing is the careful specification 
and analysis of the program objectives which produces a more meaningful 
output measure. 

To elaborate further — for the Functions Course alone — four 
priority ranked specific student— oriented objectives were established 
(step one) : 

1. To familiarize you with the terminology and basic 
techniques of Information Science. 

2. To develop your capability of identifying and defining 
problems in your professional intelligence field which 
are amenable to solution by information science techniques 
and to solve such problems at the elementary level. 

3. To improve your communications capabilities in conferring 
with information science professionals. 

4. To encourage you to pursue further the development of 
your own, and your organization's information science 
resources and capabilities. 

How well v-e accomplish these objectives in the Functions Course 
with a fixed set of resource inputs is reflected in the course 
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evaluations completed by the students, A summary of the student 
evaluation responses from the last Functions Course demonstrates 
to what degree these objectives were fulfilled for that particular 
class (step two) : 

STUDENT EVALUATION-FINDINGS 

OBJECTIVE OUTSTANDING /EXCELLENT GOOD FAIR 


1 . 

27 Students 

3 

0 


90% of the class 

10% 

— 

2. 

15 

13 

2 


50% 

43% 

7% 

3. 

21 

7 

2 


70% 

23% 

7% 

4. 

23 

6 

1 


77% 

20% 

3% 


From these findings one can conclude that all objectives were 
accomplished. Since objectives 2 and 3 rank the lowest, efforts 
should be directed toward improving those areas for the next 
offering of the Functions Course. A follow-up questionnaire, four 
to six months after the course, would further confirm the degree of 
job application accomplished by the students of each course. 

These objectives and the associated evaluation scheme used for 
the Functions Course may not be applicable for all courses conducted 
by OTR. Each course must have specif ied ; its own set of objectives 
based on the unique requirements of the customer needs to be satisfied 
by each course. How each course is conducted depends upon (1) the 
subject material to be presented, (2) the skill and knowledge of the 
assigned manpower presenting the course and (3) any procedural 

directives issued by top and/or middle management that apply 
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specifically to a particular program or course. These are the 
factors that influence not only the specification of objectives 
but also the degree to vhich the course objectives can be 
accomplished . 

OTR's Role in Establishing Requirements 

Although this process, its steps and their interrelationship 
have been explained using the Functions Course as a “real live 
intelligence application 11 the same concepts apply to any level within 
the organization* For example, the objectives that 0TR must accomplish 
have in the past depended upon the requirements that have been 
levied by "Top" management. How OTR is organized has depended upon 
(1) the entire li.st of such requirements and (2) the aggregation of 
similiar activities and/or courses (requirements) into manageable 
(school) segments. Such aggregations must provide for clearly defined 
areas of responsibility and authority which are the pre-requisites to 
accountability. To say the least I have been disappointed in the limited 
agency career development program that exists only for specific segments 
or very narrow specialties* The time has come for OfR to become more 
aggressive not only in career development training but also in much 
of the currently named “component 11 training* The training that OTR 
conducts for the Directorate of Operations and the Directorate of 
Intelligence I consider to be basically component training. In order 
to effect batter utilization of training assets by OTR, OTR must 
effectively plan, organize and control the total training effort of 

- 7 - 
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the agency. Planning sets the stage, organizing sets the resources 
and controlling sets the degree to which the organization will 
continue to survive. Planning determines a schedule of what is to 
be accomplished, Organizing determines the resources to be invested 
and how to accomplish the planned schedule and controlling determines 
how well the plan was accomplished. Since control requires a 
measured comparison of what was accomplished to what was planned to 
be accomplished, there can be no control without a succinct plan. 

The degree or amount of control required is dependent upon the 
professional capabilities of the personnel. In this regard OTR must 
decide what role it is to play in Agency Training.. This really means 
a restatement of the objectives to be accomplished, time-phased in a 
reasonable balance between assets, resources and time for accomplishment. 
X am reminded of the little boy who was asked by his father (a world 
reknowned Production Control expert) what he wanted for his birthday 
next month. The toy replied "a baby brother." His dad responded 
with "that’s impossible son." To which the boy quickly replied 
"you have taught me all of the principles and practices of effective 
production control, just put more men on the job". 

Once the objectives are carefully specified and analyzed, then 
OTR can determine if the present organization is properly structured 
to accomplish this priority-ranked list of objectives. The only 
definitive principles of organization structure that apply are that 
(1) each organization falls somewhere between the extremes of being 
functionally organized and being product or service organized and 

- 8 - 

Approved For Release 2003/04/29 : CIA-RDP83-00058R0001 001 20007-6 



Approved For Rel^ffee 2003/04/29 : Cl A-RDP83-00058R0(j<5tb01 20007-6 


(2) like-activities should be grouped into manageable segments so 
that the specified objectives of such "grouped'' activities can be 
responsibly accomplished. 

If OTR is to exert more influence in career development, then 
OTR must write career development plans (programs) in conjunction 
with each responsible directorate. These career development programs 
then become the focal point of the OTR program planning and the 
subsequent course planning. The existence of career development 
plans does not require OTR to be organized along the same career 
structure. In fact, an OTR element such as the Information Science 
Program can cut across a number of career development plans by 
providing segments of instruction or courses that fit into various 
career progression patterns. Other such examples include the IWA 
introductory orientation, the mid-career and management courses and 
the senior seminar. 

If OTR recognizes a need or void in training then a course or 
program should be developed and presented. Demand for such a program 
can be generated through effective advertising campaigns. We should 
look to the methods used in successful marketing research and advertising 
campaigns and apply such tactics to researching the need and promoting 
newly developed programs. ‘ : 

To determine how well each course is meeting its objectives, I 
offer as an example (appendix A) the form used in the Functions Course. 

As a follow-up questionnaire, I offer the form (appendix B) designed for 
the Functions Course. To determine the allocation of manpovrer/time 

assets, I offer the Faculty/Staff Activity Report (appendix C) as the 
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basic time allocation, data collection device. One can thus relate 
resource expenditures to the benefits provided to our customers — 
the students and to those responsible for review of career development. 

Some Observations 

It is often charged that such measurement sets up biases in 
decision-making by concentrating on costs and ignoring intangibles 
and human factors which cannot be quantified. Or conversely by 
naively attempting to put numbers on such imponderable elements 
thereby misleading the decision-maker. Such thinking often forces 
personnel to play games. For example (in the evaluation of personnel 
who (1) present lectures or (2) participate as guest lecturers in 
other courses.) If I knew that my performance was to be evaluated 
on (1) the number of students who were lectured (2) the number of 
my lecture hours and (3) the ratio of students per lecture hours 
then I would make certain that my lecture hours per course were at 
a maximum and that I lectured only to large student-groups or classes. 

I could so maximize my performance measures that I would always be 
number one on this performance roster. « 

However sincere these critics may be — they reflect a complete 
misunderstanding of the relevent issue. And sometimes they simply 
reflect the chagrin that particular pet projects may not show up 
well under such measurement schemes. 

Program Performance measurement does require a systematic analysis 
(the means) of program proposals and decisions, concentrating on those 
particular decisions (the ends) which have inherent budgetary 
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consequences * Please note that systematic analysis does not have to 
be quantitative and is not co-extensive with quantitative analysis. 

The word "analyze" does not have the same meaning as the words 
"quantify" or "measure" although analysis often includes some form 
of measurement. Management by Objectives, the PPB process or whatever 
the current "in" title of a good management philosopy is, all of these 
concepts seek to subject to a systematic analysis both the tangible 
and intangible elements of a program decision. We live in a world 
that must make decisions often using limited or meagre information. 
This is more akin to the European Style of Management which opts 
for a decision as contrasted with the American Style which opts for 
"adequate" information before making the decision. In any event 
let us not become so Management by Objectives (MBO) oriented that 
we become manager s who can*t make decisions unless the plan calls 
for a decision. 
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CLASS: 

INTRODUCT ION. The Staff is concerned with the quality of its educational 
program. Your constructive comments are solicited and will be used as part 
pf the basis for improving the ability of future presentations of tiie 
functions Course to meet the information science training requirements of 
user organizations. 

1. What is your overall Impression of this course? 


Outstanding Excellent Good Fair Poor 

Did we fulfill each of our course objectives for you? Indicate below: 

(a) To familiarize you with the terminology and basic techniques 
of Information Science. 


Ou tstanding_ 
(b) 


Excel 1 ent 


Good 


Fair 


Poor 


To develop your capabil ity of identifying and defining problems 
in your professional intelligence field which' are amenable to 
solution by information science techniques and to solve such 
problems at the elementary level. 


Outstanding Excellent Good Fair Poor 


(c) To improve your communications capabilities in conferring with 
Information science professionals. 


Putstanding Excellent Good Fair Poor 

(d) To encourage you to pursue further the development of your own, 
and your organization's information science resources and 
capabilities. 


Outstanding Excellent Good Fair Poor 


(e) Do you believe the course content is compatible with the course 
objectives? Yus No 

Discuss this compatibility and describe any changes that you 
think would increase this compatibility. 


Approved For Release 2003/04/29 : CIA-RDP83-00058R0001 001 20007-6 



4. Would you reconvene! this course to a co-worker? 
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5. What is your job title and what are your major tasks? 


6. What are your personal reason(s) for attending this course? 


7. Reflecting on your learning from the course what percent of the course 
do you feel will contribute or be of use to you? Please identify by 
entering one (x) check for each line. 

a. Immediate Use 

25% C'ol 7b% Tcu% 

b. Long-range Use 

2 UZ 50% 757 Tool 


8. Please rote the course in light of your answers to Questions #4. #5, 
#6 and #7. 


Outstanding 


Excellent 


Good 


Fair 


Poor 


9. As a follow-up to your end-of-course evaluation, any subsequent 

comments based on work experience at your home station would be most 
welcome and helpful in updating course relevance. Would you be 
interested in accompl ishing a post-graduate questionnaire four months 
after completion of this Course? 


Yes No 


3 
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3. Reflect on your learning from the course this week. For each subject element listed please indicate 
your personal reaction to the appropriate space, 
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SUBJECT ELEMENT 

I can see no 
application for 
this element in 
my work. 

I am aware of 
a few appli- 
cations for this 
element. 

I am aware 
of many 
applications 
for this 
element in 
my work. 

I r d like to incor- 
porate the infor- 
mation &c tech- 
niques from this 
element in my 
work. 

I will 'certainly 
incorporate 
the infor- 
mation and 
techniques 
from this 
element in my 
w o r k , 

Elementary System 
Concepts 






Basic Programming 






Statistics 






Library Programs .. . 






DELPHI 






Decision Trees 






Network Analysis 






Intelligence Problems - 

making use of 
these new techniques 
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Reflect on your 

your personal reaction to the appropriate space. 


listed please indicate 


SUBJECT ELEMENT 

■ 

I can see no 
application for 
this element in 
i my work. 

| 

| I am aware of 
a few appli- 
j cations for this 
element. 

I am aware 
of many 
applications 
for this 

element in 
my work. 

I’d like to incor- 
porate. the infor- 
mation & tech- 
niques from this 
element in my 
work. 

I will certainly : 
incorporate 
the infor- 
mation and 
techniques 
from this 

element in 
my \vork. A 

r obability 



■ 



inear Programming 






■asic Programming 



i 



ingle Correlation &r 
Regression 






library Programs 

i 

I 

i 




DELPHI 




1 


J r obability Assessment 






ntelligence Problems - 

making use of these 
new techniques 

a f 
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SUBJECT ELEMENT 


I can see no 
application for 
this element in 
mv work. 


I am aware of 
a few appli- 
cations for this 
element. 


I am aware 
of many 
applications 
for this 
element in 
my work. 


I f d like to incor- 
porate the infor- 
mation & tech- 
niques from this 
element in my . 
work. 


I will certain!! 
, incorporate | 
the inf or- j 

mation and | 
techniques | 
from this 
element in my 


ormation Storage And 
retrieval 


cision Theory 


hrary Programs 



elligence Problems - 

making use of these 
new techniques 
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SUBJECT ELEMENT 

I can see no 
application for 
this element in 
my work. 

I am aware of 
a few appli- 
cations for this 

element. 

I am aware 
of many 
applications 
for this 

element in 
my work. 

I’d like to incor- 
porate the infor- 
mation & tech- 
niques from this 
element in my 
work. 

I will certainly 
incorporate 
the infor- 
mation and 
techniques 
from this 
element in 
mv work. 

Retrieval Operations 





— 

Search Strategy 






3 a ye s ian A na 1 ys i s 






Semantic Distortion 






File Construction 




i 


MIS/PPB 






Human Factors 


! 




Modelling &c 

Simulation 






PERT 






Intelligence Problems - 

making use of these 
. 2 u w tcchni a u e s 

Af 
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NAHE: WEEK OF: T0_ 

This form is simple to maintain if done each day. Indicate the 
amount of time spent performing any activity listed. (Indicate 
others if necessary.) An activity is considered significant if 
it requires more than 15 minutes of your time to do it. In the 
case of short duration jobs, i.e., filing, merely record the 
approximate total time spent in that activity during the day. 
turn in to branch chief by Friday noon each week. 


ADMINISTRATION : 

Management : 

letter/ report-research 
letter/report-preparation 
letter/ report-coordinating 

Brief ings-preparing 
Brief ings-presenting 
Brief ings-at tending 

Meetings-preparation 

Meetings-attending 

Telephone 

Clerical : 

typing 

filing 

reproduction 

meetings 

telephone 

training 


CURRICULUM : 

Instruction : 

Research/ study 
Lesson preparation 
Lesson presentation 
Lesson attendance 
Coordinat ing 


Mon. 

Tue. 

Wed. 

Thur. 

Fri. 

O.T. 




































.. 

































































































































’ 
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CURRICULUM ACTIVITY CONTI) . 

Schedulin g: 

preparation 

coordinating 

Stud ent Advising/Evaluating 

Faculty Advisor Time 
Seminar Evaluation 
Paper/report evaluation 
Briefing evaluation 
Counseling 

Record administration 

Exercises : 

Development 
Preparation for 
Participation 

SECURITY : 

Security indoctrination/ training 
PROFESSIONALISM: 

Professional development 
Proficiency maintenance 


IRAVEL/LEAVE : 

Travel/TDY 

Leave 
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